Papers with pre-training framework
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
Semantic-based Pre-training for Dialogue Understanding (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models are weak in understanding the main semantic meaning of a dialogue context. |
| Approach: | They propose a semantic-based framework that leverages explicit semantic knowledge to capture the core semantic information in dialogues during pre-training. |
| Outcome: | The proposed model is superior to existing models on chit-chats and task-oriented dialogues. |
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023.emnlp-main)
Copied to clipboard
| Challenge: | et al., 2022) argue that the current models for drug discovery lack the ability to integrate molecules, proteins, and natural language. |
| Approach: | They propose a framework that integrates biological knowledge with chemical knowledge and natural language associations. |
| Outcome: | The proposed framework shows superior performance across a wide range of tasks. |
Integrating Structural Semantic Knowledge for Enhanced Information Extraction Pre-training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-training methods focus on exploiting textual knowledge, which limits scalability and versatility of resulting models. |
| Approach: | They propose a pre-training framework that integrates structural semantic knowledge via contrastive learning. |
| Outcome: | The proposed framework outperforms state-of-the-art pre-training methods across multiple tasks. |
Knowledge Inheritance for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
| Challenge: | Existing large-scale pre-trained language models are mainly trained from scratch individually, ignoring that many well-taught PLMs are available. |
| Approach: | They propose a pre-training framework called knowledge inheritance and propose auxiliary supervision to efficiently learn larger PLMs. |
| Outcome: | The proposed framework can be used to train large-scale language models with huge parameters and a large dataset can be adapted to domain adaptation and knowledge transfer. |
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical results show that CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. |
| Approach: | They introduce a pre-training framework that unifies cross-lingual and cross-modal pre-trained models with shared architectures and objectives. |
| Outcome: | The proposed framework outperforms the state-of-the-art in two multi-lingual datasets and two multilingual image-text retrieval datasets. |
FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing pre-trained models need fine-tuning on tens of thousands of examples to achieve good results. |
| Approach: | They propose a framework that leverages pre-trained text-to-text models and aligns them with their pre-training framework. |
| Outcome: | The proposed framework outperforms the XLM-Roberta-large on multiple QA benchmarks and is applicable to multilingual situations. |
Unleashing the Power of Language Models in Text-Attributed Graph (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on graph learning on text-attributed graphs have been limited by memory cost and underutilization of relationships between nodes and words. |
| Approach: | They propose a Node Representation Update Pre-training Architecture based on Co-modeling text and graph to learn representations of papers and words simultaneously. |
| Outcome: | The proposed model outperforms baselines on the ogbn-arxiv benchmark dataset. |
Pre-training Language Models for Comparative Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions. |
| Approach: | They propose a framework to pre-train language models for enhancing comparative reasoning abilities . they collect scalable data for text-based entity comparison . |
| Outcome: | The proposed framework significantly improves comparative reasoning abilities under low-resource conditions on downstream tasks. |
Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining (2023.emnlp-main)
Copied to clipboard
Jingcong Liang, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei
| Challenge: | Recent studies have discussed its capability to assist language models for various applications. |
| Approach: | They propose a structure to organize arguments using the **Hi**erarchical **Ar**gumentation **G**raph (Hi-ArG) and propose two approaches to exploit Hi-AarG, including a text-graph multi-modal model GreaseArR and a framework augmented with graph information. |
| Outcome: | The proposed structure supersedes existing language models on two argumentation tasks while incorporating graph information during further training improves vanilla language models. |